Systems, methods, and apparatuses for implementing maximum likelihood image binarization in a coded light range camera

ABSTRACT

In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing maximum likelihood image binarization in a coded light range camera. For instance, a depth camera is described having therein a projector to project a collection of planes, each at a different angle of projection, onto a scene via a plurality of coded pattern images, each of the coded pattern images having encoded therein via a plurality of stripes, the angle of projection for the plane of projection within which the respective coded pattern image is projected; a detector to capture the plurality of coded pattern images from the scene; a processing component to adjust for ambient illumination and reflection properties of the scene; in which the processing component to further output a bit value for each pixel in the captured plurality of coded pattern images and to output a sub-pixel offset for the pixels positioned upon transitions of the plurality of stripes in the captured plurality of coded pattern images; a decoder to decode each of the plurality of coded pattern images and to adjust the decoded plurality of coded image patterns based on the sub-pixel offsets to determine the angle of projection for the corresponding plane of projection; and a triangulator to determine a position of an object in the scene based on an intersection of the determined angle of projection for the corresponding plane of projection with a known ray emanating from the detector that detected the plurality of the coded pattern images from the scene. Other related embodiments are disclosed.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

The subject matter described herein relates generally to the field ofimage capture devices such as cameras, and more particularly, tosystems, methods, and apparatuses for implementing maximum likelihoodimage binarization in a coded light range camera.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also correspond toembodiments of the claimed subject matter.

Conventional cameras capture a single image from a single optical focalpoint and are enabled to capture pixels corresponding to an object in ascene, but in so doing, such cameras lose the depth information forwhere within the scene that object is positioned in terms of depth ordistance from camera.

Conversely, stereo cameras have two or more lenses, each with a separateimage sensor, and the two or more lenses allow the camera to capturethree-dimensional images through a process known as stereo photography.With such conventional stereo cameras, triangulation is used todetermine the depth to an object in a scene using a process known ascorrespondence. Correspondence presents a problem, however, ofascertaining which parts of one image captured at a first of the lensescorrespond to parts of another image, captured at a second of thelenses. That is to say, which elements of the two photos correspond toone another as they represent the same portion of an object in thescene, such that triangulation may be performed to determine the depthto that object in the scene.

Given two or more images of the same three-dimensional scene, taken fromdifferent points of view via the two or more lenses of the stereocamera, correspondence processing requires identifying a set of pointsin one image which can be correspondingly identified as the same pointsin another image by matching points or features in one image with thecorresponding points or features in another image.

This processing, however, is computationally intensive and thereforerequires additional computing hardware to process higher quality imageryor necessitates a delay between image capture and correspondenceprocessing completion from which the depth to an object may bedetermined and thus eliminates the possibility of real-time imageprocessing as is required with moving video. Moreover, complexities arefurther introduced through variables such as movement of the camera, theelapse of time and/or movement of objects in the photos, variability inlighting conditions, and so forth. Still further, it may be that thescene from which the depth to an object is to be measured is nearlyfeatureless, and as such, the correspondence processing cannot ascertainpoints which match to one another in the images. Consider for instancecapturing images of a white wall or a featureless scene and trying toidentify matching points within the scene. The correspondence processingwill likely fail to identify sufficient correspondence points betweenthe images thus making triangulation ineffective.

The present state of the art may therefore benefit from the systems,methods, and apparatuses for implementing maximum likelihood imagebinarization in a coded light range camera as is described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way oflimitation, and will be more fully understood with reference to thefollowing detailed description when considered in connection with thefigures in which:

FIG. 1 illustrates an exemplary architecture in accordance with whichembodiments may operate;

FIG. 2 illustrates another exemplary architecture in accordance withwhich embodiments may operate;

FIG. 3 illustrates another exemplary architecture in accordance withwhich embodiments may operate;

FIG. 4 illustrates another exemplary architecture in accordance withwhich embodiments may operate;

FIG. 5A depicts a flow diagram having exemplary architecture andcircuitry in accordance with which embodiments may operate;

FIG. 5B illustrates another exemplary architecture in accordance withwhich embodiments may operate;

FIG. 6A is a flow diagram illustrating a method for implementing maximumlikelihood image binarization in a coded light range camera inaccordance with the described embodiments;

FIG. 6B is an alternative flow diagram illustrating a method forimplementing maximum likelihood image binarization in a coded lightrange camera in accordance with the described embodiments;

FIG. 7A illustrates an exemplary tablet computing device with a cameraenclosure housing the depth camera assembly in accordance with describedembodiments;

FIG. 7B illustrates an exemplary hand-held smartphone with a cameraenclosure housing the depth camera assembly in accordance with describedembodiments;

FIG. 7C is a block diagram of an embodiment of tablet computing device,a smart phone, or other mobile device in which touchscreen interfaceconnectors are used; and

FIG. 8 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system, in accordance with one embodiment.

DETAILED DESCRIPTION

Described herein are systems, apparatuses, and methods for implementingmaximum likelihood image binarization in a coded light range camera. Forinstance, a depth camera is described having therein a projector toproject a collection of planes, each at a different angle of projection,onto a scene via a plurality of coded pattern images, each of the codedpattern images having encoded therein via a plurality of stripes, theangle of projection for the plane of projection within which therespective coded pattern image is projected; a detector to capture theplurality of coded pattern images from the scene; a processing componentto adjust for ambient illumination and reflection properties of thescene; in which the processing component is to further output a bitvalue for each pixel in the captured plurality of coded pattern imagesand to output a sub-pixel offset for the pixels positioned upontransitions of the plurality of stripes in the captured plurality ofcoded pattern images; a decoder to decode each of the plurality of codedpattern images and to adjust the decoded plurality of coded imagepatterns based on the sub-pixel offsets to determine the angle ofprojection for the corresponding plane of projection; and a triangulatorto determine a position of an object in the scene based on anintersection of the determined angle of projection for the correspondingplane of projection with a known ray emanating from the detector thatdetected the plurality of the coded pattern images from the scene.

It shall be understood that the term “ray” is in reference to themathematical object and the term “ray” is not in reference to anyprojection (e.g., radiating light) by the depth camera, which arereferred to herein as projections, coded pattern images, planesprojected by the depth camera, etc. A ray is a half of a line, that isto say, a straight line starting at an initial point and extending toinfinity in a direction and having no curvature, for which there is onlyone dimension, namely length, without width nor depth. As used herein,the mathematical object “ray” is therefore distinguished from a “beam”which is an optical term.

A single image of a scene captured via a single camera will have nodepth information whatsoever for an object because given an imaginaryray originating from the focal point of the camera and extending to apoint in the image, such as a pixel located on the object of the scene,it is impossible to determine where upon the ray that pixel is located,and thus, it is unknown where the object is positioned within the scene.

There is ambiguity therefore with respect to the position of the objectin the scene. Triangulation enables the recovery of this depthinformation so as to identify the position of an object in a scene, bydetermining where two rays intersect, one from each of two stereocameras.

Conventionally, depth cameras operated on the basis ofcorrespondence-based triangulation from stereo cameras to estimatedepth. Such conventional stereo cameras seek to determine a depth fromthe position of the camera to, ideally, every pixel of an object withina captured image. With such cameras, triangulation operates bycalculating the intersection between two rays, each of the two raysoriginating from two different optical systems.

When an image is captured by such conventional cameras, it is necessaryto compute correspondence for the pixels within the captured imagebefore triangulation may be applied to determine depth, distance, orrange to any given pixel of an object within the image.

There are multiple problems with such a conventional approach. One suchdrawback is the sheer computational intensity of computingcorrespondence. As image fidelity increases due to improvements in imagecapture technology the number of pixels in any given image increasesdramatically. Such increases represent difficult processing demands intheir own right, from the storage of the resulting image to the busprocessing speed for which the data of a captured image may be takenfrom an image capture circuit and stored on non-transitory memory,necessitating that massive amounts of data be captured and stored inbetween every single still image or for every single frame of a movingimage.

Such problems are only compounded when dealing with three-dimensionalimagery. Correspondence calculations require identifying matching pixelsfor a given object in a scene amongst multiple captured images at thedifferent optical systems, such that a subsequent triangulationoperation may be performed. Conventional cameras which rely uponcorrespondence derived triangulation therefore are a legitimate andappropriate means by which to determine depth, but at greatcomputational expense, which translates directly in to increased productcosts due to the necessity to provide sufficient processing, memory, andbus technology within such a system.

Moreover, conventional systems utilizing correspondence derivedtriangulation suffer from a kind of depth blindness in the presence of ascene which is void of detail. Consider for instance such a conventionalcamera which captures left and right stereo images of a white wall. Sucha system cannot calculate correspondence for such a featureless scene,and as such, is simply unable to perform the subsequent triangulation.Though a white while may be an extreme example, it is quite common forsmaller areas of a captured scene to have portions that lack sufficientdetail with which to compute correspondence, due to, for instance,lighting, distance, a lack of pixel density, and so forth, and thisinability for such conventional cameras to compute correspondence forthose sub-areas of the captured scene result in significant error in thedepth computations and degradation of depth determining performance.

Therefore, the depth camera as is described herein introduces an activecomponent into the scene to produce active stereo imagery or coded lightimagery of the scene. Through this approach it is possible to remove oneof the cameras and replace it instead with a projector. For instance,the projector may then be utilized to project one-dimensional code ontothe scene, such as a sequence of patterns which may subsequently becaptured by the remaining camera and decoded.

In accordance with one embodiment, the depth camera provides an activecoded light triangulation system, utilizing an infrared (IR) projector,an IR camera, and a Red, Green, Blue (RGB) camera. Such a system mayinclude coded light range cameras operating by projecting a sequence ofone-dimensional binary (“black” and “white”) patterns onto the scene,such that the produced binary code encodes the angle of the projectionplane. The binarizer produces binary code bits from the projectedsequence of the one-dimensional binary patterns, which the decoder thentranslates and correlates to an angle as per the encoded information,followed by the triangulator which reconstructs the depth bytriangulating the intersection of a ray emanating from the camera withthe now known angle of the plane emanating from the projector.

In certain embodiments, an image of the projected pattern is comparedagainst a reference fully illuminated image and a reference fullyun-illuminated image, and the corresponding bit of the binary code iscomputed at each pixel. There is a need to differentiate between blackand white pixels for bit coding of a captured image, and detecting asub-pixel transition between black and white pixels of those pixelswhich lie upon a transition, which allows the system to achieve higherdepth reconstruction accuracy over those systems utilizing simplebinarization based on a threshold which yields less accurate results aserrors in the coding may result in significant accuracy loss in 3D depthestimation.

According to a particular embodiment, the system determines the pixelcode of a pixel (e.g., whether the pixel is black or white) based onrelative intensity of a group of neighboring pixels using a “template”of a transition as captured by the camera/detector. The template may beconfigured to take into account the point spread function of theprojector, lens and camera. Using a matching score function, e.g., loglikelihood taking into account various noise sources, the systemdetermines a matching between an area around the pixel and an area ofthe template thus yielding superior range and better accuracy comparedto conventional solutions which are based on point-wise binarization orcorrelation.

Improved systems and methods are described for performing thebinarization process. For instance, in order to improve thesignal-to-noise ratio, the binarization of a single pixel is based on alocal environment of pixels, and the actual shape of the rising andfalling edges of the template binary pattern while accounting for theprojector and camera point spread function which is assumed to be known.The captured image of the pattern is normalized using the two referenceimages, the fully illuminated image and the fully un-illuminated image,in order to undo the surface reflection properties and angle andrange-related attenuation. The value of the binarized pixel is computedby matching sub-pixel shifts of the template, possibly at slightlydifferent orientations, to the normalized image, and selecting the shiftproducing the best match.

In such a way, a sub-pixel location of the pattern edge is calculatedalongside with the binary image. In experiments, straightforwardmatching criteria such as correlation do not correctly account for thesources of noise present in a depth camera system, and in particular,fail to correctly account for Poisson-distributed shot noise of thesensor whose variance is proportional to the signal strength.

A more principled maximum likelihood estimator is therefore describedherein to improve the estimations. In addition, likelihood values areutilized to compute the confidence of each pixel, in which theconfidence is subsequently aggregated across the bit planes of the codeand is used at the decoding stage.

In the following description, numerous specific details are set forthsuch as examples of specific systems, languages, components, etc., inorder to provide a thorough understanding of the various embodiments. Itwill be apparent, however, to one skilled in the art that these specificdetails need not be employed to practice the embodiments disclosedherein. In other instances, well known materials or methods have notbeen described in detail in order to avoid unnecessarily obscuring thedisclosed embodiments.

In addition to various hardware components depicted in the figures anddescribed herein, embodiments further include various operations whichare described below. The operations described in accordance with suchembodiments may be performed by hardware components or may be embodiedin machine-executable instructions, which may be used to cause ageneral-purpose or special-purpose processor programmed with theinstructions to perform the operations. Alternatively, the operationsmay be performed by a combination of hardware and software.

Embodiments also relate to an apparatus for performing the operationsdisclosed herein. This apparatus may be specially constructed for therequired purposes, or it may be a general purpose computer selectivelyactivated or reconfigured by a computer program stored in the computer.Such a computer program may be stored in a computer readable storagemedium, such as, but not limited to, any type of disk including floppydisks, optical disks, CD-ROMs, and magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs,magnetic or optical cards, or any type of media suitable for storingelectronic instructions, each coupled with a computer system bus. Theterm “coupled” may refer to two or more elements which are in directcontact (physically, electrically, magnetically, optically, etc.) or totwo or more elements that are not in direct contact with each other, butstill cooperate and/or interact with each other.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear as set forth in thedescription below. In addition, embodiments are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the embodiments as described herein.

Any of the disclosed embodiments may be used alone or together with oneanother in any combination. Although various embodiments may have beenpartially motivated by deficiencies with conventional techniques andapproaches, some of which are described or alluded to within thespecification, the embodiments need not necessarily address or solve anyof these deficiencies, but rather, may address only some of thedeficiencies, address none of the deficiencies, or be directed towarddifferent deficiencies and problems which are not directly discussed.

FIG. 1 illustrates an exemplary architecture in accordance with whichembodiments may operate. In particular, depicted are both a depth camera110 and a scene 195, the scene 195 having an object therein, for whichthe depth camera 110 has determined the depth to the object 190 of thescene 195 as indicated by element 185.

Within depth camera 110 there is a projector 115, a detector 120, aprocessing component 125 (e.g., also referred to herein as processingcircuitry which may include, for instance, one or more CPUs, memory,busses, FPGAs, etc.), a decoder 130, and a triangulator.

In accordance with a particular embodiment, such a depth camera 110includes the projector 115 to project a collection of planes, each at adifferent angle of projection, onto a scene 195 via a plurality of codedpattern images 170, each of the coded pattern images 170 having encodedtherein via a plurality of stripes, the angle of projection for theplane of projection within which the respective coded pattern image 170is projected. The depth camera 110 further includes a detector 120 tocapture the plurality of coded pattern images 170 from the scene 195 anda processing component 125 to adjust for ambient illumination andreflection properties of the scene 195.

According to such an embodiment, the processing component 125 is tofurther output a bit value 126 for each pixel in the captured pluralityof coded pattern images 170 and additionally is to output a sub-pixeloffset 127 for the pixels positioned upon transitions 128 of theplurality of stripes in the captured plurality of coded pattern images170.

According to one embodiment, each sub-pixel offset 127 represents anamount of pixel shift relative to a center of the pixel for which thetransition occurs between one of the plurality of stripes.

The depicted depth camera still further includes a decoder 130 to decodeeach of the plurality of coded pattern images 170 and to adjust thedecoded plurality of coded pattern images 170 based on the sub-pixeloffsets 127 to determine the angle of projection for the correspondingplane of projection.

The triangulator 135 of the depicted depth camera 110 is to determine aposition of an object 190 in the scene 195 based on an intersection ofthe determined angle of projection for the corresponding plane ofprojection with a geometric ray originating from the detector 120 thatdetected the plurality of the coded pattern images 170 from the scene195.

In accordance with such an embodiment, the processing component 125outputs a bit value 126 for each and every pixel of the captured scene195 and separately outputs the sub-pixel offsets 127 for only a sub-setof the pixels in the scene 195, and specifically, outputs the sub-pixeloffsets 127 for those pixels which lie upon a transition 128 of one ofthe stripe which make up each of the coded pattern images 170. Forinstance, some of the pixels in the scene 195 will be entirely within orentirely upon one of the stripes, and as such, no sub-pixel offset 127will be provided, although a bit value will be provided as bit valuesare provided for each and every pixel. Other pixels will positioned beentirely off of and away from the stripes which make up the codedpattern images 170, and thus, like the pixels positioned entirely upon astripe, no sub-pixel offset 127 will be provided. Conversely, thosepixels which bridge an edge or transition of any of the stripes, suchthat a portion of the pixel is located upon the stripe and a portion ofthe pixel is located off of the stripe, thus being a pixel on atransition 128, will have both a bit value 126 output by the processingcomponent and additionally a sub-pixel offset 127 output by theprocessing component.

FIG. 2 illustrates another exemplary architecture in accordance withwhich embodiments may operate. In particular, depth camera 210 isdepicted having specifically an infrared projector 215 to project thecollection of planes onto the scene 295 using the plurality of codedpattern images 270 (via rows, columns, or both). In place of thepreviously described detector there is an infrared camera 220 whichoperates as a detector to detect the coded pattern images 270 from thescene 295.

As depicted here, the depth camera 210 further includes an RGB (Red,Green, Blue) camera in accordance with one embodiment. For instance, insuch an embodiment, the projector of the depth camera is an infrared 215to project the collection of planes onto the scene in an infrared lightrange and the detector is an infrared camera 220 to capture theplurality of coded pattern images from the scene in the infrared lightrange and the depth camera further includes an RGB camera 255 to captureRGB images of the scene in a visible light range.

Further depicted is a binarizer/normalizer 225 having processingcircuitry therein to normalize for lighting conditions of the scene andto binarize and output the bit values 226 for the pixels of the sceneand the sub-pixel offsets 227 for those pixels which lie upon atransition of one of the plurality of stripes in the captured pluralityof coded pattern images 270.

Each plane is projected onto the scene 295 at a different angle. If theangle of each plane projected on to the scene 295 or as projected ontothe object 290 of the scene is known, then it is possible tomathematically derive the position of the object 290 in the scene 295based on the intersection between the ray originating at the detector ofthe depth camera and the projected plane.

However, it is necessary to encode the angle or position of theintersecting plane within the scene 295; hence the use of the stripes ofthe coded pattern images 270 which take the form of a binary code,referred to as “Gray code” or “Gray patterns.” Both rows and columns maybe utilized to form the coded pattern images 270 via the stripes.

Thus, it is according to described embodiments, the projector projectsone-dimensional code onto the scene via a sequence of patterns that aresubsequently captured by the detector 120 or infrared camera 220 anddecoded by the decoder 130.

In accordance with one embodiment, the projector which is to project theplurality of coded pattern images onto the scene operates within thenon-visible light range utilizing both an infrared projector 215 toproject the plurality of coded pattern images in the form of1-dimensional stripes for each of the coded pattern images, the1-dimensional stripes projected within the plurality of coded patternimages corresponding to a multi-bit binary code projected by theinfrared projector.

According to another embodiment, the multi-bit binary code constitutesan N-bit Gray code projected by the infrared projector 215 via N codedpattern images 270, the most significant bit of the N-bit Gray codeprojected as the 1-dimensional stripes within a first of the N codedpattern images and a least significant bit of the N-bit Gray codeprojected as the 1-dimensional stripes within a last of the N codedpattern images 270.

According to one embodiment, any single pixel position captured from theplurality of coded pattern images 270 projected onto the scene 295corresponds to a sequence of 0s or 1s in the N-bit Gray code based onthe pixel being illuminated or un-illuminated within the scene at thesingle pixel position during each of the plurality of coded patternimages captured.

FIG. 3 illustrates another exemplary architecture in accordance withwhich embodiments may operate. In particular, depth camera 310 isdepicted having specifically the projector 315 to project the collectionof planes 316 onto the scene 395 using the plurality of coded patternimages and a detector 320 to detect the coded image patterns from thescene 395.

According to a particular embodiment, the decoder 130 is to determinewhich plane 316 the point of an object 390 is lying within the scene 395based on a sequence of 0s or 1s in a N-bit Gray code projected onto thescene 395 as 1-dimensional stripes within the N coded pattern images.For instance, in such an embodiment, the triangulator 135 is todetermine the position of the object 390 in the scene by an intersection322 of the determined plane 316 with the geometric ray 321 originatingfrom the detector that detected the plurality of the coded patternimages from the scene 395. In such an embodiment, the position of theobject 390 in the scene 395 is determined without computingcorrespondence between the plurality of coded pattern images projectedonto the scene by the projector 315 and the plurality of coded patternimages captured from the scene by the detector 320.

In accordance with one embodiment, the projector 315 is to project theplurality of coded pattern images onto the scene constitutes an infraredprojector to project one dimensional code onto the scene via a sequenceof coded patterns to be captured by the infrared detector and decoded bythe decoder 130 to determine the angle of projection for each of thecollection of planes which the triangulator 135 receives as an input totriangulate intersections 322 between the planes of the projectedplurality of coded pattern images on the scene with the geometric ray321 originating from the infrared detector which is to capture theplurality of coded pattern images on the scene and further in which theone dimensional code specifies the angle of projection for each plane ofthe collection of planes projected onto the scene by the projector 315.

As noted above, the projector 315 (whether infrared or otherwise) usescoded light to encode the angles of each of the collection of planesprojected onto the scene via the coded pattern images. The coded lightprovides a series of one dimensional stripes that look like an N-bitbinary code.

Take for example an 8-bit code to be projected onto the scene. Theprojector 315 projects eight such patterns, one for each bit of thecode, each of the patterns projected onto the scene encoding one bit,starting from most significant bit to the least significant bit of theexemplary 8-bit binary code. Depending on the position of the object390, there is a sequence of zeros and ones (0s and 1s), determinablebased on whether any given pixel is an illuminated pixel or anun-illuminated pixel.

According to one embodiment, the depth camera of includes abinarizer/normalizer 335 as implemented by processing circuitry or theprocessing component as depicted at FIGS. 1 and 2. In such anembodiment, the normalizer and binarizer units 335 reduce noise andinterference in the plurality of coded pattern images captured by thedetector before the triangulator 135 determines the position of theobject 390 in the scene. In such an embodiment, the normalizer andbinarizer units 335 further convert the plurality of coded patternimages into binary information, the binary information specifying atleast the angle of projection for each plane in the collection of planesprojected onto the scene as adjusted by the decoder 130 based on thesub-pixel offsets 127, as depicted at FIGS. 1 and 2.

Extracting the sequence of bits from the 8-bit code projected onto thescene the binarizer 335 in conjunction with the decoder 130 is able toidentify on which of the projected collection of planes 316 any givenpoint on the object 390 lies and by intersecting 322 that plane with aray 321 originating at the focal point of the detector 320, the positionand depth of that point, and thus the position of the object 390, isdeterminable via the triangulator 135.

In accordance with another embodiment, the triangulator 135 is todetermine a distance from the depth camera to every one of a pluralityof pixel locations on the object 390 as captured by the detector 320based on the intersection 322 between the plane at the determined angleof projection within which the pixel location on the object is capturedand the geometric ray 321 originating from the detector at the samelocation. In such an embodiment, the position of the object 390 in thescene is found in 3-dimensional space relative to the depth camera basedon the determined distance to the plurality of pixel locations on theobject determined by the triangulator 135.

In accordance with one embodiment, the triangulator 135 which is todetermine the position of the object in the scene constitutes thetriangulator determining the position of the object 390 in the scene 395without computing correspondence between the plurality of coded patternimages projected onto the scene by the projector 315 and the pluralityof coded pattern images captured from the scene by the detector 320.

Because the intersection 322 is determined for the point between anidentified projected plane and the intersecting 322 ray 321 from thedetector 320, it not necessary to compute correspondence for pixels ofmultiple focal point images, and thus, the process is significantly lesscomputationally intensive and thus much faster in terms of processingtime and responsiveness, especially in consumer oriented platforms forwhich battery consumption, heat, and the cost of processing componentsis a significant area of concern for manufacturers and retailers.

The binarizer 335 then proceeds to decode the information bydetermining, for each pixel captured by the detector 320, whether thesequence of stripes represented a 1 or a 0 so as to output theappropriate bit-value for that pixel.

FIG. 4 illustrates another exemplary architecture in accordance withwhich embodiments may operate. In particular, depth camera 410 isdepicted having specifically the projector 415 to project the collectionof planes 499 onto the scene 495 using the plurality of coded patternimages and a detector 420 to detect the coded image patterns from thescene 495. The projector 415 includes lens assembly 416 having optics497 embedded therein and detector 420 includes lens assembly 421, alsohaving optics embedded therein, although not depicted here.

Given the exemplary 8-bit code, there are 256 different planes encodedonto the scene. Around every one of the collection of planes projectedonto the scene there exists some uncertainty. Take for example, a fieldof view of 64 degrees and by dividing the field of view approximatelyequally there will be a resulting 256 different regions, each with anuncertainty of plus or minus 0.25 degrees. For this area of uncertaintythere are a finite number of bits for the code. Estimating depth evenjust a few meters from the camera becomes increasingly problematic dueto the lack of resolution in which the angle of the plane is inflated ormagnified by a factor of several thousand. Due to the manner oftriangulation for the intersections described above, the calculationsare very sensitive to errors in the position of the plane. Also depictedare the binarizer 435 and normalizer 436.

According to one embodiment, the binarizer 335 of the processingcomponent estimates positions of transition from 0 to 1 and 1 to 0 forthe plurality of coded pattern images in a sub-pixel range of thedetector 320 where the transitions occur for the pixels positioned uponany of the plurality of stripes in the captured plurality of codedpattern images. In such an embodiment, the estimated positions provide ahigher resolution of the captured coded image patterns than is encodedby the values alone of the plurality of stripes.

According to another embodiment, the binarizer 335 determines a binaryvalue for each pixel captured by the detector 320 within the pluralityof coded pattern images and outputs the bit value for each pixel. Insuch an embodiment, the binarizer 335 further determines the pixel shiftfor each pixel having a transition from 0 to 1 or 1 to 0 in theplurality of coded pattern images, in which the pixel shift represents ashift relative to a center of the pixel having the transition. Thebinarizer 335 further outputs the sub-pixel offset for the pixelspositioned upon transitions of the plurality of stripes in the capturedplurality of coded pattern images and provides the binary value for eachpixel and the determined pixel shifts as an input to the decoder 130.

According to a particular embodiment, the resolution of depthinformation in a subset of pixels of each projected coded pattern imageis proportional to angular resolution of the detector and is independentfrom the quantity of bits encoded within the plurality of coded patternimages.

Stated differently, the resolution of depth information in the subset ofthe output image does not depend on the number of code bits. Rather,greater resolution of depth information is derived from an increasedangular resolution of the detector without a corresponding increase in aquantity of bits encoded within the plurality of coded pattern imagesprojected by the projector.

Depth resolution may therefore be improved beyond that which is feasiblethrough projected bit code when taken alone. For instance, at infiniteresolution the plurality of coded pattern images projected onto thescene appear as a step function from zero to one and from one to zero.Notwithstanding the blur 498, each bit may be escalated at the positionof transitions from zero to one or from one to zero with sub-pixelresolution which is much more accurate than that which is yielded by thebit values for each one of the pixels alone. Greater resolution is thusdeterminable for at least portions of the image. Thus, for each pixel,not only is the bit value output, but for those pixels located orpositioned upon where the transitions occur, there is also output somevalue of pixel shift with respect to, for example, the pixel center.This information, the bit value and the pixel shift for those pixels ona transition, is then fed to a processing component through a pipelineand on to a triangulator which then determines the position and depthinformation for the object in the scene.

Conversely, if the depth camera were to be limited by the number of bitsactually projected via the plurality of coded image patterns, then everybit required to communicate information, such as the angle ofprojection, requires that another pattern be projected onto the scene,which therefore requires projection and detection time to convey andcapture the information which then in turn induces additionalsensitivity to movement and motion within the scene, for instance, dueto lower permissible frame rates, due to the increased demand onprojecting and detecting the coded image patterns from the scene. Theseadditional projection and detection times are overhead and thus detractfrom the underlying content that a user is actually interested inseeing.

Therefore, other camera solutions which use active coded lighttechnology are extremely limited by the number of bits they are able toproject. With a high resolution camera the resolution of depthinformation using conventional techniques is thus limited by theprojected bit information whereas utilizing the techniques describedherein, including providing the pixel shift information in the sub-pixelrange for those pixels which lie upon a transition, the resolution ofdepth information is greatly enhanced by the pixel size of the highresolution camera and is not limited by the number of bits projected bythe projector.

In accordance with certain embodiments, optics 497 of the projector 415or the detector 420, or both the projector and the detector, at leastpartially blur 498 the projected collection of planes 499 at the edgesof each plane. In such an embodiment, a binarizer 435 of the processingcomponent adjusts for the blur 498 prior to the triangulator 135determining the position of the object 490 in the scene 495.

The normalizer 436 does not undo the blur, but rather, it undoes theambient light and the reflection coefficient of the scene 495, known as“albedo,” which are unknown to the normalizer 435 in advance. To adjustfor the blur 498 at the edges of each of the collection of planes withinthe region of uncertainty, the overall point spread function of theentire optical 497 system including the projector 415 and the detector420 is made part of the templates.

In an alternative embodiment, blur 498 is accounted for at the binarizer435 and is not based on the reflection coefficients and ambient light.The binarizer 435 operates to attain a higher resolution of the code atthose points within the region of uncertainty than would otherwise befeasible through the number of code bits alone.

According to one embodiment, a quantity of illumination in each pixel inthe projected and captured coded pattern images is unknown in advanceand is based on a yet to be determined distance to the pixel at theobject 490, unknown reflection characteristics of the object at thepixel's position on the object, and unknown ambient illumination. Insuch an embodiment, a normalizer 436 of the processing componentcompensates for the unknowns based on a comparison of projected andcaptured coded pattern images to a projected and captured fullyilluminated image and a captured un-illuminated image having only theambient light of the scene present therein.

According to another embodiment, a first reference image of all 1s forevery pixel is established from the projected and captured fullyilluminated image 491 and a second reference image of all 0s for everypixel is established from the captured un-illuminated image 492 havingonly the ambient light of the scene present therein. In such anembodiment, the normalizer 436 is to determine an offset illumination ofeach pixel from the first reference image, and determine the codeamplitude from the difference in illumination between the first andsecond reference images and further in which the normalizer 436 is tonormalize the captured coded pattern images by subtracting the offsetfrom every pixel of the captured coded pattern images and dividing theresult by the code amplitude for every pixel of the captured codedpattern images.

Resolution decreases as the distance to the object increases because theamount of light reflected decays as the square of the distance to theobject and due also to the reflection properties including thereflective coefficient, called albedo, the amount of light the objectreflects back. Additionally complicating the matter is the manner bywhich the light is reflected back, with one extreme being a mirroredreflection versus the reflective properties of a white piece of paper,resulting in diffuse reflection. In reality, objects within the scenewill reflect light back to the depth camera somewhere in between the twoextremes, causing the code projected onto the scene to modulateaccording to the unknown reflection characteristics of the scene and theobjects present in the scene.

Projecting a fully illuminated frame provides two reference frames fromwhich the system may normalize all the rest of the coded pattern imagesby subtracting the offset due to the ambient illumination dividing bythe distance between zero and one in every pixel in order to adjust forthe ambient lighting and reflection characteristics of the objects inthe scene.

Signal Formation Model:

According to the described embodiments, use of the fully illuminatedimage and the fully un-illuminated image constitute a signal formationmodel defined as follows:

Let I₀ and I₁ denote the un-illuminated (dark) image and fullyilluminated (white) images, respectively and let I_(p) be the image ofthe known projected pattern. For each pixel in I, determine whether thepattern was black (0) or white (1) at that location. If the transitionfrom 0 to 1 (a rising edge) or from 1 to 0 (a falling edge) occurs atthat pixel, a sub-pixel estimation of the transition location shall becalculated. Translating each of the three images denoted collectively asI_(*) with * representing the 0, 1, or p from DN to photoelectronsaccording to: Y_(*)(x)=γ max {I_(*)(x)−dc, 0}, where dc is the knownsensor A/D offset and where γ is the known pixel convergence gain. Thistranslation is required since unlike I_(*), Y_(*) admit Poissonstatistics.

The un-illuminated image can be modeled as: Y₀(x)=C(x)+N(x) where C(x)is the contribution of the ambient illumination having Poissondistribution with the rate λ_(amb), and N is stationary zero-mean randomnoise due to pixel dark current, quantization, and read-out. Both noisesare approximated as Gaussian, having: Y₀(x)≈λ_(a)(x)+N(0,σ²+λ_(a)(x)),statistically independent at different pixels, with σ² denoting thevariance of N.

The fully-illuminated image has the same contribution of the ambient, towhich the contribution of the projected signal is added: Y₁(x)=A(x)+N(x)where: A(x) has Poisson distribution with the rate: λ_(a)+λ_(p)(1+S), Sbeing a zero-mean noise modeling the laser speckle which is static intime but varying in space, and in which N is additional system noise asY₀.

As before, all noises are approximated as Gaussian, yielding:Y₁(x)≈λ_(a)(x)+λ_(p)(x)+N(0,σ²+λ_(a)(x)+λ_(p)(x)+v²λ_(p) ²(x)), with v²representing the variance of the speckle noise.

The pattern image is formed similarly to Y₁ with the exception that theprojector contribution is modulated by the projected pattern P(x)normalized between 0 (no illumination) and 1 (full illumination),yielding:Y _(p)(x)≈λ_(a)(x)+λ_(p)(x)P(x)+N(0,σ²+λ_(a)(x)+λ_(p)(x)P(x)+v ²λ_(p)²(x)V ²(X)).

Depth determination is sensitive to noise, and thus, the system adjustsfor a variety of noise sources as detailed above. Therefore, inaccordance with one embodiment, a binarizer 435 of the processingcomponent is to adjust for noise in the captured coded pattern imagesbefore the triangulator determines the position of an object 490 in thescene 495.

According to one embodiment, the binarizer 435 is to adjust for noiseselected from one or more of the following noise types: shot noise orcounting noise resulting from a photon hitting a pixel and depositingcharge with the pixel at an uncertain probability according to Poissondistribution in which the variance increases with the code amplitude;noise originating from circuitry of the detector due to quantization ofthe captured coded pattern images into a digital signal, wherein thenoise is invariant with the strength of the signal; and speckle noiseoriginating from a light source of the infrared projector creatinginterference patterns via coherent illumination of the scene.

According to one embodiment, the binarizer 435 is to output the bitvalue for each pixel in the captured plurality of coded pattern images.In such an embodiment, the binarizer further determines a confidencevalue for each pixel and aggregates the confidence value of each pixelacross each of the plurality of coded pattern images projected in thecollection of planes and in which the decoder then receives as input theaggregated confidence value determined for each pixel and decodes theplurality of coded pattern images based at least in part on theaggregated confidence values.

Maximum Likelihood Estimation:

In a particular embodiment, likelihood values are used to compute theconfidence of each pixel, the confidence values of the pixels beingsubsequently aggregated across the bit planes of the code used at thedecoding stage, as follows:

Let pixel location x be fixed, dropping the dependence on it forconvenience of notation, and further assume that λ_(a) and λ_(p) areknown, although in practice, their estimates are likely to be provided.

Given the measurement y=Y_(p)(x), the log-likelihood of P∈[0,1] beingthe projected pattern P(x) is given by:

${L( y \middle| P )} = {{\log\;{P( {Y_{p} = P} )}} = {{const} - {\frac{( {\lambda_{a} + {\lambda_{p}P} - y} )^{2}}{2( {\sigma^{2} + \lambda_{a} + {\lambda_{p}P} + {v^{2}\lambda_{p}^{2}V^{2}}} )}.}}}$

According to one embodiment, the normalizer 436 is to estimate thevalues of the ambient signal λ_(a) and the signal amplitude λ_(p) foreach pixel based on a patch of neighboring pixels surrounding the pixel.

In order to estimate λ_(a) and λ_(p), assume that these two parametersare approximately constant in a patch P around x, and estimate them byaveraging:

${{\hat{\lambda}}_{a} = {\frac{1}{P}{\sum\limits_{x \in P}\;{y_{0}(x)}}}},$where y_(*)(x) denotes the measurements of the random variablesY_(*)(x). Similarly,

${\hat{\lambda}}_{p} = {\max{\{ {{\frac{1}{P}{\sum\limits_{x \in P}\;( {{y_{1}(x)} - {y_{0}(x)}} )}},0} \}.}}$

Using these estimates, the log likelihood of observing the patch of ygiven the pattern P is:

${{L( y \middle| P )} = {\sum\limits_{x \in P}\;\frac{( {{\hat{\lambda}}_{a} + {{\hat{\lambda}}_{p}{P(x)}} - {y(x)}} )^{2}}{\sigma^{2} + {\hat{\lambda}}_{a} + {{\hat{\lambda}}_{p}{P(x)}} + {{\hat{\lambda}}_{p}^{2}( {{v^{2}{V^{2}(x)}} + \eta^{2}} )}}}},$where the variance parameter η² is added to account for model noisestemming from inaccuracies in the modeling of the actual projectedpattern.

According to one embodiment, the binarizer 435 is to output the bitvalue for each pixel in the captured plurality of coded pattern imagesand the binarizer further determines the bit value of each pixel basedon a patch of neighboring pixels surrounding the pixel by selecting oneof a plurality of patch templates which matches the patch of neighboringpixels with a better resilience to noise and the binarizer then outputsthe bit value for each pixel in the captured plurality of coded imagepatterns based on the selected one patch template for the pixel.

For instance, the binarizer may select one of a plurality of patchtemplates P(x) which matches the patch of neighboring pixels y(x) basedon which of the templates provides a greatest degree of accuracy or adetermined accuracy over a pre-determined threshold or a determinedimprovement in noise over a threshold.

According to one embodiment, the binarizer 435 is to compute a sum ofthe squared differences based on a difference between the pixels in eachpatch in the plurality of coded pattern images captured from the sceneand the pixels in each of the plurality of template patches,

${\sum\limits_{x \in P}\;{{w(x)}( {{{\hat{\lambda}}_{p}{P(x)}} - {y(x)}} )^{2}}},$where terms of the sum of the squared differences are weighted byweights w(x) based on signal strength to identify which one of aplurality of templates matches a patch of neighboring pixels with agreatest degree of accuracy; and further in which the binarizer is tooutput the bit value for each pixel in the captured plurality of codedimage patterns based on the selected one patch template for the pixel.

According to one embodiment, the binarizer 435 is to compute a weightedcorrelation

${\sum\limits_{x \in P}\;{{w(x)}{P(x)}{y(x)}}},$based on a sum of products between the pixels in each patch in theplurality of coded pattern images captured from the scene and the pixelsin each of the plurality of template patches, wherein terms of the sumof the products are weighted by weights w(x) based on signal strength toidentify which one of a plurality of templates matches a patch ofneighboring pixels with a greatest degree of accuracy; and further inwhich the binarizer is to output the bit value for each pixel in thecaptured plurality of coded image patterns based on the selected onepatch template for the pixel.

Mathematically, the computation of the sum of the squared differencesand the computation of the weighted correlation based on a sum ofproducts between the pixels are similar, but not identical, and each mayprovide certain benefits over the other for any given implementation.

Maximum Likelihood (ML) Binarizer:

Binarization of Y_(p) based on the maximum likelihood estimatordescribed above is performed as follows:

First, the knowledge of the projected pattern is assumed (aone-dimensional code), which is expressed by two one-dimensionalfunctions p^(±)(x) with the transition (respectively, rising and fallingedges) located exactly at x=0. A parametric family of two-dimensionaltemplates is formed by letting for each pixel x=(x,y) in the patch toassume the value: P_(δ,0) ^(±)(x,y)=p^(±)(x−δ−θy), where the parameter δaccounts for the shift of the pattern, while θ approximates itsrotation, e.g., due to the lens radial distortion.

For each location x in the image, a patch P around it is used toestimate {circumflex over (λ)}_(a) and {circumflex over (λ)}_(p). Whileθ is treated as a nuisance parameter, the shift δ is actually estimatedusing maximum likelihood:

${\hat{\delta}(x)} = {\arg{\max\limits_{\delta}{\max\limits_{\theta}{\max{\{ {{\sum\limits_{x \in P}\;{L( {y(x)} \middle| {P_{\delta,\theta}^{-}(x)} )}},{\sum\limits_{x \in P}\;{L( {y(x)} \middle| {P_{\delta,\theta}^{+}(x)} )}}} \}.}}}}}$

In practice, the likelihood is discretized on a grid of δ and θ. If therising edge template P⁺ is selected and δ<0, or the falling edge oftemplate P⁻ is selected and δ≥1, the pixel is assigned the value of 0.Similarly, if the rising edge template P⁺ is selected and δ>1, or thefalling edge template P⁻ is selected and δ≤1, the pixel is assigned thevalue of 0. Otherwise, if δ∈[0,1], a transition is signaled in the pixelx, and its sub-pixel location is encoded by {circumflex over (δ)}(x).Furthermore, the maximum value of the likelihood function serves as theconfidence of the estimation.

FIG. 5A depicts a flow diagram having exemplary architecture andcircuitry in accordance with which embodiments may operate. Specificallydepicted are memory 581 and IR camera 562, leading to normalizer 582 inwhich I₀ and I₁ denote the fully un-illuminated (dark) image and fullyilluminated (white) images, respectively and I_(p) is the image of theknown projected pattern. Refer to the signal formation model above.

Templates 561 are depicted as an input to the Maximum LikelihoodBINarizer (MLBIN) 583 as are {circumflex over (λ)}_(a) and {circumflexover (λ)}_(p) and Y_(p) from the normalizer 582. MLBIN 583 providesconfidence, parameter δ and B_(p) to decoder 584. Decoder 584 providesconfidence, parameter δ and X_(p) to code filter 585 resulting in X_(p)refined being passed to triangulation unit 586 which receives as inputsX_(p) and X_(c) as well as calibration parameters and temperature datafrom the thermal correction circuit 563 based on temperature sensor(s)564. Triangulation 586 passes Z to depth filters 587 resulting in Zrefined.

FIG. 5B illustrates another exemplary architecture in accordance withwhich embodiments may operate. In particular, depth camera 510 isdepicted having specifically the projector 515 and detector 520 therein,as well as a lens assembly 516 and associated optics 597 of projectorand a lens assembly 521 of detector 520. Binarizer 535 and normalizer536 are again depicted, herein within the processing component 525. Alsodepicted are decoder 130 and triangulator 135.

In accordance with a particular embodiment, the depth camera 510 furtherincludes a first temperature sensor 517 to measure a first temperatureat the projector 515 and a second temperature sensor 522 to measure asecond temperature at the detector 520. In such an embodiment, theprocessing component 525 is to receive the first and second temperatures523 as an input to adjust calibration parameters 524 of the triangulatorused to determine the position of the object in the scene, theadjustment to the calibration parameters 524 based on temperaturedependent changes to optical 597, mechanical, and electronic componentsin the projector 515 and the detector 520.

In accordance with a particular embodiment, the depth camera implementscorrections, sometimes referred to “summer correction” to account forchanges in temperature of the depth camera, of the components of thedepth camera, of the operating conditions within which the depth camerafunctions, or some combination thereof. Calibration parameters 524 ofthe depth camera are affected by temperature and as such, changes intemperature may reduce the precision and accuracy of the depth cameradue to physical changes in the optics, thus negatively affecting depthmeasurements performed by the depth camera. Therefore, the temperaturesensors and temperature readings are utilized by the depth camera toadjust the calibration parameters utilized by the triangulator inperforming the depth measurements and specifically used to determine thedepth to a given object observed in the scene. Such changes affectingthe optics of the depth camera may be no more than one part in athousand, yet even very small discrepancies can be amplified over thedistance to an object in the scene and as such, even very smallcorrections can improve the performance and accuracy of the depthcamera.

In accordance with a particular embodiment, temperature sensors areembedded in a lens of the depth camera, such as within the projectorlens assembly or the detector or camera lens assembly, and the depthcamera receives temperature data from the lens assembly which is thenutilized by the triangulator to adjust the calibration parameters. Inaccordance with another embodiment, temperature sensors are affixed to aprinted circuit board near or co-located with the projector or thedetector.

In accordance with a particular embodiment, a correction circuit 511receives as input one or more temperatures measured via a correspondingone or more temperature sensors at or near the optics of the depthcamera, such as the optics of the detector and projector, and thecorrection circuit 511 outputs either an adjustment to the calibrationparameters utilized by the triangulator or outputs adjusted correctionparameters for use by the triangulator. For instance, the adjustmentcircuit may apply a so-called summer correction to the calibrationparameters which are then provided as an input to the triangulator.

FIG. 6A is a flow diagram illustrating a method 600 for implementingmaximum likelihood image binarization in a coded light range camera inaccordance with the described embodiments. FIG. 6B is an alternativeflow diagram illustrating a method 601 for implementing maximumlikelihood image binarization in a coded light range camera inaccordance with the described embodiments.

Some of the blocks and/or operations listed below for methods 600 and601 are optional in accordance with certain embodiments. The numberingof the blocks presented is for the sake of clarity and is not intendedto prescribe an order of operations in which the various blocks mustoccur. Methods 600 and 601 may be performed by processing logic that mayinclude hardware (e.g., circuitry, dedicated logic, programmable logic,microcode, etc.), software (e.g., instructions run on a processingdevice) to perform various operations such as projecting, capturing,adjusting, outputting, decoding, triangulating, normalizing, binarizing,controlling, analyzing, collecting, monitoring, executing, presenting,interfacing, receiving, processing, determining, triggering, displaying,etc., in pursuance of the systems and methods as described herein. Forexample, depth cameras 110, 210, 310, 410, and 510 as depicted at FIGS.1 through 5, the smart phone or tablet computing devices as depicted atFIGS. 7A, 7B, and 7C, or the machine 800 at FIG. 8, may implement thedescribed methodologies.

With reference first to method 600 of FIG. 6A, operations begins atblock 605 with processing logic for projecting via a projector, acollection of planes, each at a different angle of projection, onto ascene via a plurality of coded pattern images, each of the coded patternimages having encoded therein via a plurality of stripes, the angle ofprojection for the plane of projection within which the respective codedpattern image is projected.

At block 610, processing logic captures, via a detector, the pluralityof coded pattern images from the scene.

At block 615, processing logic adjusts for ambient illumination andreflection properties of the scene via a processing component of thedepth camera.

At block 620, processing logic outputs from the processing component abit value for each pixel in the captured plurality of coded patternimages and outputting a sub-pixel offset for the pixels positioned upontransitions of the plurality of stripes in the captured plurality ofcoded pattern images.

At block 625, processing logic decodes each of the plurality of codedpattern images and adjusting the decoded plurality of coded imagepatterns based on the sub-pixel offsets to determine the angle ofprojection for the corresponding plane of projection.

At block 630, processing logic triangulates a position of an object inthe scene based on an intersection of the determined angle of projectionfor the corresponding plane of projection with a geometric rayoriginating from the detector that captured the plurality of the codedpattern images from the scene.

In accordance with a particular embodiment of method 600, the processingcomponent includes a binarizer and the method further determines abinary value for each pixel captured by the detector within theplurality of coded pattern images and output the bit value for eachpixel. In such an embodiment, the method still further determines apixel shift for each pixel having a transition from 0 to 1 or 1 to 0 inthe plurality of coded pattern images, the pixel shift representing ashift relative to a center of the pixel having the transition output thesub-pixel offset for the pixels positioned upon transitions of theplurality of stripes in the captured plurality of coded pattern images;and provides the binary value for each pixel and the determined pixelshifts as an input to a decoder.

Further optional processing may take place to normalize the image,binarize the image, implement temperature sensitive summer correctionsfor the image, and so forth, consistent with the embodiments as aredescribed herein.

In accordance with another embodiment there is a non-transitory computerreadable storage medium having instructions stored thereupon that, whenexecuted by a processor of a depth camera, the instructions cause thedepth camera to perform operations including: projecting a collection ofplanes, each at a different angle of projection, onto a scene via aplurality of coded pattern images, each of the coded pattern imageshaving encoded therein via a plurality of stripes, the angle ofprojection for the plane of projection within which the respective codedpattern image is projected; capturing the plurality of coded patternimages from the scene; adjusting for ambient illumination and reflectionproperties of the scene via a processing component of the depth camera;outputting from the processing component a bit value for each pixel inthe captured plurality of coded pattern images and outputting asub-pixel offset for the pixels positioned upon transitions of theplurality of stripes in the captured plurality of coded pattern images;decoding each of the plurality of coded pattern images and adjusting thedecoded plurality of coded image patterns based on the sub-pixel offsetsto determine the angle of projection for the corresponding plane ofprojection; and triangulating a position of an object in the scene basedon an intersection of the determined angle of projection for thecorresponding plane of projection with a geometric ray originating fromthe detector that captured the plurality of the coded pattern imagesfrom the scene.

With reference now to method 601 of FIG. 6B, operations begins at block650 with processing logic capturing as an input, a captured patternimage and two reference images, a first image as a fully illuminatedprojected image as an “all white” reference image and a second image asa fully un-illuminated projected image as an “all black” referenceimage.

At block 655, processing logic normalizes the captured pattern imageusing a window of neighboring pixels around each pixel of the capturedpattern image to determine background level (C) based the all blackreference image and background plus signal level (A) based on the allwhite reference image, in which B=max(0, A−C).

At block 660, processing logic determines the binarized image and atransition map which indicates transitions of the captured pattern imageat the sub-pixel level, specifying within the transition map thelocation of transition with respect to the beginning of each pixel whichlies upon a transition, in which the determination of whether or not apixel is black or white is based on a neighborhood of pixels surroundingthe pixel.

At block 665, processing logic determines a transition delta whichmaximizes the likelihood that the captured image resulted from thetransition delta, in which the transition is determined based on atemplate of transition which is configured to take into account thespread function of the optics of the system including the system'sprojector which projects the images onto a scene and the detector whichcaptures the images from the scene.

At block 670, processing logic accounts for noise in the systemutilizing each or both of the binarization and transition likelihoodtaking into account various noise sources.

At block 675, processing logic determines the confidence level of apixel based on the noise variance where for SNR (Signal-to-Noise-Ratio)of a pixel that is too low, the pixel is valued to be of low confidenceor determining the confidence level based on the value of the costfunction at the maximized point, or both.

At block 680, processing logic applies the confidence level of thepixels to select which pixels are used to perform the computation inwhich full likelihood estimation for low confidence pixels are outrightbypassed to save power, time, and processing resources. For instance,any pixel having a confidence at or below a pre-determined threshold maybe deemed too low of confidence, and as such, not used in thecomputation.

FIG. 7A illustrates an exemplary tablet computing device 701 with acamera enclosure 746 housing the depth camera assembly 799 in accordancewith described embodiments. FIG. 7B illustrates an exemplary hand-heldsmartphone 702 with a camera enclosure 746 housing the depth cameraassembly 799 in accordance with described embodiments.

For instance, according to the described embodiments, the depth cameraassembly 799 having the necessary optics (e.g., lenses) of the projector715, detector 716, as well as the normalizer/binarizer 725, decoder 726,and triangulator 735 therein, as described previously, is integratedwithin a hand-held smartphone 702 or tablet computing device 701 as acamera body for the depth camera assembly 799.

In accordance with one embodiment, the hand held smartphone 702 ortablet computing device 701 having a touchscreen interface 745integrated therein forms the camera body to which the depth cameraassembly 799 is integrated or installed.

FIGS. 7A and 7B depict the tablet computing device 701 and the hand-heldsmartphone 702 each having a circuitry integrated therein as describedin accordance with the embodiments. As depicted, each of the tabletcomputing device 701 and the hand-held smartphone 702 include atouchscreen interface 745 and an integrated processor 711 in accordancewith disclosed embodiments.

For example, in one embodiment, a tablet computing device 701 or ahand-held smartphone 702, includes a display unit which includes atouchscreen interface 745 for the tablet or the smartphone and furtherin which memory and an integrated circuit operating as an integratedprocessor are incorporated into the tablet or smartphone, in which theintegrated processor is operable in conjunction with the depth cameraassembly 799 and its components and circuitry as described herein; thetablet or smartphone and its processing components being furtheroperable to perform image projection, image capture and image processingincluding adjustments and triangulation to determine depth informationto an object of a scene. In one embodiment, the integrated circuitdescribed above or the depicted integrated processor of the tablet orsmartphone is an integrated silicon processor functioning as a centralprocessing unit (CPU) and/or a Graphics Processing Unit (GPU) for atablet computing device or a smartphone.

FIG. 7C is a block diagram 703 of an embodiment of tablet computingdevice, a smart phone, or other mobile device in which touchscreeninterface connectors are used. Processor 710 performs the primaryprocessing operations. Audio subsystem 720 represents hardware (e.g.,audio hardware and audio circuits) and software (e.g., drivers, codecs)components associated with providing audio functions to the computingdevice. In one embodiment, a user interacts with the tablet computingdevice or smart phone by providing audio commands that are received andprocessed by processor 710.

Depth camera assembly 799 is depicted as communicably interfaced to theprocessor 710 and peripheral connections 780. Depth camera assembly 799includes the projector 798 and detector 797.

Display subsystem 730 represents hardware (e.g., display devices) andsoftware (e.g., drivers) components that provide a visual and/or tactiledisplay for a user to interact with the tablet computing device or smartphone. Display subsystem 730 includes display interface 732, whichincludes the particular screen or hardware device used to provide adisplay to a user. In one embodiment, display subsystem 730 includes atouchscreen device that provides both output and input to a user.

I/O controller 740 represents hardware devices and software componentsrelated to interaction with a user. I/O controller 740 can operate tomanage hardware that is part of an audio subsystem 720 and/or displaysubsystem 730. Additionally, I/O controller 740 illustrates a connectionpoint for additional devices that connect to the tablet computing deviceor smart phone through which a user might interact. In one embodiment,I/O controller 740 manages devices such as accelerometers, cameras,light sensors or other environmental sensors, or other hardware that canbe included in the tablet computing device or smart phone. The input canbe part of direct user interaction, as well as providing environmentalinput to the tablet computing device or smart phone.

In one embodiment, the tablet computing device or smart phone includespower management 790 that manages battery power usage, charging of thebattery, and features related to power saving operation. Memorysubsystem 760 includes memory devices for storing information in thetablet computing device or smart phone. Connectivity 770 includeshardware devices (e.g., wireless and/or wired connectors andcommunication hardware) and software components (e.g., drivers, protocolstacks) to the tablet computing device or smart phone to communicatewith external devices. Cellular connectivity 772 may include, forexample, wireless carriers such as GSM (global system for mobilecommunications), CDMA (code division multiple access), TDM (timedivision multiplexing), or other cellular service standards). Wirelessconnectivity 774 may include, for example, activity that is notcellular, such as personal area networks (e.g., Bluetooth), local areanetworks (e.g., WiFi), and/or wide area networks (e.g., WiMax), or otherwireless communication.

Peripheral connections 780 include hardware interfaces and connectors,as well as software components (e.g., drivers, protocol stacks) to makeperipheral connections as a peripheral device (“to” 782) to othercomputing devices, as well as have peripheral devices (“from” 784)connected to the tablet computing device or smart phone, including, forexample, a “docking” connector to connect with other computing devices.Peripheral connections 780 include common or standards-based connectors,such as a Universal Serial Bus (USB) connector, DisplayPort includingMiniDisplayPort (MDP), High Definition Multimedia Interface (HDMI),Firewire, etc.

FIG. 8 illustrates a diagrammatic representation of a machine 800 in theexemplary form of a computer system, in accordance with one embodiment,within which a set of instructions, for causing the machine/computersystem 800 to perform any one or more of the methodologies discussedherein, may be executed. In alternative embodiments, the machine may beconnected (e.g., networked) to other machines in a Local Area Network(LAN), an intranet, an extranet, or the public Internet. The machine mayoperate in the capacity of a server or a client machine in aclient-server network environment, as a peer machine in a peer-to-peer(or distributed) network environment, as a server or series of serverswithin an on-demand service environment. Certain embodiments of themachine may be in the form of a personal computer (PC), a tablet PC, aset-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, a web appliance, a server, a network router, switch orbridge, computing system, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines (e.g., computers) that individually or jointly execute a set(or multiple sets) of instructions to perform any one or more of themethodologies discussed herein including implementing maximum likelihoodimage binarization in a coded light range camera.

The exemplary computer system 800 includes a processor 802, a mainmemory 804 (e.g., read-only memory (ROM), flash memory, dynamic randomaccess memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM(RDRAM), etc., static memory such as flash memory, static random accessmemory (SRAM), volatile but high-data rate RAM, etc.), and a secondarymemory 818 (e.g., a persistent storage device including hard disk drivesand a persistent database and/or a multi-tenant databaseimplementation), which communicate with each other via a bus 830. Mainmemory 804 includes software 822 and an depth processing 824functionality which is interfaced with the triangulation circuit 823capable of performing triangulation processing without the need forcorrespondence calculation amongst a set of multiple images captured.Main memory 804 and its sub-elements are operable in conjunction withprocessing logic 826 and processor 802 to perform the methodologiesdiscussed herein.

Processor 802 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 802 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processor 802 may alsobe one or more special-purpose processing devices such as an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), a digital signal processor (DSP), network processor, or thelike. Processor 802 is configured to execute the processing logic 826for performing the operations and functionality which is discussedherein including interfacing to the depth camera and/or performingprocessing on behalf of such a depth camera.

The computer system 800 may further include a network interface card808. The computer system 800 also may include a user interface 810 (suchas a video display unit, a liquid crystal display (LCD), touch screen,or a cathode ray tube (CRT)), an alphanumeric input device 812 (e.g., akeyboard), a cursor control device 814 (e.g., a mouse), and a signalgeneration device such as an integrated speaker 816. The computer system800 may further include peripheral device 836 (e.g., wireless or wiredcommunication devices, memory devices, storage devices, audio processingdevices, video processing devices, etc.).

The secondary memory 818 may include a non-transitory machine-accessibleor computer readable storage medium 831 on which is stored one or moresets of instructions (e.g., software 822) embodying any one or more ofthe methodologies or functions described herein. The software 822 mayalso reside, completely or at least partially, within the main memory804 and/or within the processor 802 during execution thereof by thecomputer system 800, the main memory 804 and the processor 802 alsoconstituting machine-readable storage media. The software 822 mayfurther be transmitted or received over a network 820 via the networkinterface card 808.

While the subject matter disclosed herein has been described by way ofexample and in terms of the specific embodiments, it is to be understoodthat the claimed embodiments are not limited to the explicitlyenumerated embodiments disclosed. To the contrary, the disclosure isintended to cover various modifications and similar arrangements aswould be apparent to those skilled in the art. Therefore, the scope ofthe appended claims should be accorded the broadest interpretation so asto encompass all such modifications and similar arrangements. It is tobe understood that the above description is intended to be illustrative,and not restrictive. Many other embodiments will be apparent to those ofskill in the art upon reading and understanding the above description.The scope of the disclosed subject matter is therefore to be determinedin reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A depth camera comprising: a projector to projecta collection of planes, each at a different angle of projection, onto ascene via a plurality of coded pattern images, each of the coded patternimages having encoded therein via a plurality of stripes, the angle ofprojection for the plane of projection within which the respective codedpattern image is projected; a detector to capture the plurality of codedpattern images from the scene; a processing component to adjust forambient illumination and reflection properties of the scene; theprocessing component to further output a bit value for each pixel in thecaptured plurality of coded pattern images and to output a sub-pixeloffset for the pixels positioned upon transitions of the plurality ofstripes in the captured plurality of coded pattern images; a decoder todecode each of the plurality of coded pattern images and to adjust thedecoded plurality of coded image patterns based on the sub-pixel offsetsto determine the angle of projection for the corresponding plane ofprojection; and a triangulator to determine a position of an object inthe scene based on an intersection of the determined angle of projectionfor the corresponding plane of projection with a geometric rayoriginating from the detector that detected the plurality of the codedpattern images from the scene.
 2. The depth camera of claim 1: whereinthe projector comprises an infrared projector to project the collectionof planes onto the scene in an infrared light range; wherein thedetector comprises an infrared camera to capture the plurality of codedpattern images from the scene in the infrared light range; and whereinthe depth camera further comprises an RGB (Red, Green, Blue) camera tocapture RGB images of the scene in a visible light range.
 3. The depthcamera of claim 1, further comprising: a first temperature sensor tomeasure a first temperature at the projector; a second temperaturesensor to measure a second temperature at the detector; and wherein theprocessing component to receive the first and second temperatures as aninput to adjust calibration parameters of the triangulator used todetermine the position of the object in the scene, the adjustment to thecalibration parameters based on temperature dependent changes tooptical, mechanical, and electronic components in the projector and thedetector.
 4. The depth camera of claim 1, wherein the triangulator todetermine the position of the object in the scene comprises thetriangulator to determine the position of the object in the scenewithout computing correspondence between the plurality of coded patternimages projected onto the scene by the projector and the plurality ofcoded pattern images captured from the scene by the detector.
 5. Thedepth camera of claim 1: wherein the triangulator to determine adistance from the depth camera to every one of a plurality of pixellocations on the object as captured by the detector based on theintersection between the plane at the determined angle of projectionwithin which the pixel location on the object is captured and thegeometric ray originating from the detector at the same location; andwherein the position of the object in the scene is found in3-dimensional space relative to the depth camera based on the determineddistance to the plurality of pixel locations on the object determined bythe triangulator.
 6. The depth camera of claim 1, wherein the projectorto project the plurality of coded pattern images onto the scenecomprises: an infrared projector to project one dimensional code ontothe scene via a sequence of coded patterns to be captured by theinfrared detector and decoded by the decoder to determine the angle ofprojection for each of the collection of planes which the triangulatorreceives as an input to triangulate intersections between the planes ofthe projected plurality of coded pattern images on the scene with thegeometric ray originating from the infrared detector which is to capturethe plurality of coded pattern images on the scene; and wherein the onedimensional code specifies the angle of projection for each plane of thecollection of planes projected onto the scene by the projector.
 7. Thedepth camera of claim 1, wherein the projector to project the pluralityof coded pattern images onto the scene comprises: an infrared projectorto project the plurality of coded pattern images in the form of1-dimensional stripes for each of the coded pattern images, the1-dimensional stripes projected within the plurality of coded patternimages corresponding to a multi-bit binary code projected by theinfrared projector.
 8. The depth camera of claim 7: wherein themulti-bit binary code comprises an N-bit Gray code projected by theinfrared projector via N coded pattern images, the most significant bitof the N-bit Gray code projected as the 1-dimensional stripes within afirst of the N coded pattern images and a least significant bit of theN-bit Gray code projected as the 1-dimensional stripes within a last ofthe N coded pattern images; and wherein any single pixel positioncaptured from the plurality of coded pattern images projected onto thescene corresponds to a sequence of 0s or 1s in the N-bit Gray code basedon the pixel being illuminated or un-illuminated within the scene at thesingle pixel position during each of the plurality of coded patternimages captured.
 9. The depth camera of claim 8: wherein the decoder isto determine which plane the point of an object is lying within thescene based on the sequence of 0s or 1s in the N-bit Gray code; whereinthe triangulator is to determine the position of the object in the sceneby an intersection of the determined plane with the geometric rayoriginating from the infrared detector that detected the plurality ofthe coded pattern images from the scene; and wherein the position of theobject in the scene is determined without computing correspondencebetween the plurality of coded pattern images projected onto the sceneby the projector and the plurality of coded pattern images captured fromthe scene by the infrared detector.
 10. The depth camera of claim 1,wherein the processing component comprises: a normalizer and a binarizerunits to reduce noise and interference in the plurality of coded patternimages captured by the detector before the triangulator determines theposition of the object in the scene and to further convert the pluralityof coded pattern images into binary information, the binary informationspecifying at least the angle of projection for each plane in thecollection of planes projected onto the scene as adjusted by the decoderbased on the sub-pixel offsets.
 11. The depth camera of claim 1, whereineach sub-pixel offset represents an amount of pixel shift relative to acenter of the pixel for which the transition occurs between one of theplurality of stripes.
 12. The depth camera of claim 1, wherein optics ofthe projector or the detector, or both the projector and the detector,at least partially blur the projected collection of planes at the edgesof each plane; and wherein a binarizer of the processing componentadjusts for the blur prior to the triangulator to determine the positionof the object in the scene.
 13. The depth camera of claim 12: wherein abinarizer of the processing component is to estimate positions oftransition from 0 to 1 and 1 to 0 for the plurality of coded patternimages in a sub-pixel range of the detector where the transitions occurfor the pixels positioned upon any of the plurality of stripes in thecaptured plurality of coded pattern images; and wherein the estimatedpositions provide a higher resolution of the captured coded imagepatterns than is encoded by the values alone of the plurality ofstripes.
 14. The depth camera of claim 1, wherein the processingcomponent comprises a binarizer, wherein the binarizer is to: determinea binary value for each pixel captured by the detector within theplurality of coded pattern images and output the bit value for eachpixel; determine a pixel shift for each pixel having a transition from 0to 1 or 1 to 0 in the plurality of coded pattern images, the pixel shiftrepresenting a shift relative to a center of the pixel having thetransition; output the sub-pixel offset for the pixels positioned upontransitions of the plurality of stripes in the captured plurality ofcoded pattern images; and provide the binary value for each pixel andthe determined pixel shifts as an input to the decoder.
 15. The depthcamera of claim 1, wherein the resolution of depth information in asubset of pixels of each projected coded pattern image is proportionalto angular resolution of the detector and is independent from thequantity of bits encoded within the plurality of coded pattern images.16. The depth camera of claim 1, wherein a quantity of illumination ineach pixel in the projected and captured coded pattern images is unknownin advance and is based on a yet to be determined distance to the pixelat the object, unknown reflection characteristics of the object at thepixel's position on the object, and unknown ambient illumination; andwherein a normalizer of the processing component compensates for theunknowns based on a comparison of projected and captured coded patternimages to a projected and captured fully illuminated image and acaptured un-illuminated image having only the ambient light of the scenepresent therein.
 17. The depth camera of claim 16, wherein a firstreference image of all is for every pixel is established from theprojected and captured fully illuminated image; wherein a secondreference image of all Os for every pixel is established from thecaptured un-illuminated image having only the ambient light of the scenepresent therein; wherein the normalizer is to determine an offsetillumination of each pixel from the first reference image, and determinethe code amplitude from the difference in illumination between the firstand second reference images; and wherein the normalizer is to normalizethe captured coded pattern images by subtracting the offset from everypixel of the captured coded pattern images and dividing the result bythe code amplitude for every pixel of the captured coded pattern images.18. The depth camera of claim 1, wherein a binarizer of the processingcomponent is to adjust for noise in the captured coded pattern imagesbefore the triangulator determines the position of an object in thescene.
 19. The depth camera of claim 18, wherein the binarizer is toadjust for noise selected from one or more of the following noise types:shot noise or counting noise resulting from a photon hitting a pixel anddepositing charge with the pixel at an uncertain probability accordingto Poisson distribution in which the variance increases with the codeamplitude; noise originating from circuitry of the detector due toquantization of the captured coded pattern images into a digital signal,wherein the noise is invariant with the strength of the signal; andspeckle noise originating from a light source of the infrared projectorcreating interference patterns via coherent illumination of the scene.20. The depth camera of claim 1, wherein the processing componentcomprises a binarizer to output the bit value for each pixel in thecaptured plurality of coded pattern images; wherein the binarizerdetermines a confidence value for each pixel and aggregates theconfidence value of each pixel across each of the plurality of codedpattern images projected in the collection of planes; and wherein thedecoder receives as input the aggregated confidence value determined foreach pixel and decodes the plurality of coded pattern images based atleast in part on the aggregated confidence values.
 21. The depth cameraof claim 1, wherein the processing component comprises a binarizer tooutput the bit value for each pixel in the captured plurality of codedpattern images; wherein the binarizer determines the bit value of eachpixel based on a patch of neighboring pixels surrounding the pixel byselecting one of a plurality of patch templates which matches the patchof neighboring pixels with a better resilience to noise; and wherein thebinarizer outputs the bit value for each pixel in the captured pluralityof coded image patterns based on the selected one patch template for thepixel.
 22. The depth camera of claim 21, wherein the processingcomponent comprises a binarizer to output the bit value for each pixelin the captured plurality of coded pattern images; wherein the binarizeris to compute a sum of the squared differences based on a differencebetween the pixels in each patch in the plurality of coded patternimages captured from the scene and the pixels in each of the pluralityof template patches, wherein terms of the sum of the squared differencesare weighted based on signal strength to identify which one of aplurality of templates matches a patch of neighboring pixels with agreatest degree of accuracy; and wherein the binarizer outputs the bitvalue for each pixel in the captured plurality of coded image patternsbased on the selected one patch template for the pixel.
 23. The depthcamera of claim 21, wherein the processing component comprises abinarizer to output the bit value for each pixel in the capturedplurality of coded pattern images; wherein the binarizer is to compute aweighted correlation based on a sum of products between the pixels ineach patch in the plurality of coded pattern images captured from thescene and the pixels in each of the plurality of template patches,wherein terms of the sum of the products are weighted based on signalstrength to identify which one of a plurality of templates matches apatch of neighboring pixels with a greatest degree of accuracy; andwherein the binarizer outputs the bit value for each pixel in thecaptured plurality of coded image patterns based on the selected onepatch template for the pixel.
 24. A method in a depth camera, whereinthe method comprises: projecting via a projector, a collection ofplanes, each at a different angle of projection, onto a scene via aplurality of coded pattern images, each of the coded pattern imageshaving encoded therein via a plurality of stripes, the angle ofprojection for the plane of projection within which the respective codedpattern image is projected; capturing, via a detector, the plurality ofcoded pattern images from the scene; adjusting for ambient illuminationand reflection properties of the scene via a processing component of thedepth camera; outputting from the processing component a bit value foreach pixel in the captured plurality of coded pattern images andoutputting a sub-pixel offset for the pixels positioned upon transitionsof the plurality of stripes in the captured plurality of coded patternimages; decoding each of the plurality of coded pattern images andadjusting the decoded plurality of coded image patterns based on thesub-pixel offsets to determine the angle of projection for thecorresponding plane of projection; and triangulating a position of anobject in the scene based on an intersection of the determined angle ofprojection for the corresponding plane of projection with a geometricray originating from the detector that captured the plurality of thecoded pattern images from the scene.
 25. Non-transitory computerreadable storage media having instructions stored thereupon that, whenexecuted by a processor of a depth camera, the instructions cause thedepth camera to perform operations comprising: projecting via aprojector, a collection of planes, each at a different angle ofprojection, onto a scene via a plurality of coded pattern images, eachof the coded pattern images having encoded therein via a plurality ofstripes, the angle of projection for the plane of projection withinwhich the respective coded pattern image is projected; capturing, via adetector, the plurality of coded pattern images from the scene;adjusting for ambient illumination and reflection properties of thescene via a processing component of the depth camera; outputting fromthe processing component a bit value for each pixel in the capturedplurality of coded pattern images and outputting a sub-pixel offset forthe pixels positioned upon transitions of the plurality of stripes inthe captured plurality of coded pattern images; decoding each of theplurality of coded pattern images and adjusting the decoded plurality ofcoded image patterns based on the sub-pixel offsets to determine theangle of projection for the corresponding plane of projection; andtriangulating a position of an object in the scene based on anintersection of the determined angle of projection for the correspondingplane of projection with a known ray emanating from the detector thatcaptured the plurality of the coded pattern images from the scene.